ollama 0.1.31 -> 0.1.33 #309330
ollama 0.1.31 -> 0.1.33 #309330
Conversation
NixOSInfra
added
the
12. first-time contribution
|
I tested this (by running So, I'm against merging this into the current nixpkgs-unstable, since this is a breaking change that removes support for rocm, and breaking changes are restricted since the next stable release will be soon (#303286). I may be alright with merging this into the next unstable after nixos-24.05 has released, but until then I think it would be a bad idea to merge a breaking change. |
There was a problem hiding this comment.
Also, since it's known that rocm is broken, meta.broken should be updated accordingly, unless/until the rocm build can be fixed.
meta = {
description = "Get up and running with large language models locally";
homepage = "https://github.com/ollama/ollama";
changelog = "https://github.com/ollama/ollama/releases/tag/v${version}";
license = licenses.mit;
platforms = platforms.unix;
+ broken = enableRocm;
mainProgram = "ollama";
maintainers = with maintainers; [ abysssol dit7ya elohmeier ];
};
|
ollama 0.1.34 released |
|
If possible please move the package derivation to |
|
ollama 0.1.36 has been released |
|
ollama 0.1.37 has been released |
as far as missing dependencies, I was able to get passed this using the rocm-core package in rocmPath. There is a collision between rocmPackages-clr and clr from rocm-core, so I removed it for now to test. I also had to set the CLBlast_DIR path and append some paths to CMAKE_PREFIX_PATH at the pre-build step for this to work for me. However, this brings me to other errors, specifically with building for gfx1010. It seems like some libraries are still not built correctly. I have played around with the build script but have had no luck for now. |
|
I haven't fully tested, but I was able to get ollama to see my AMD card. It may just work? Here's what I changed from your original PR from last week: Obviously changing the default acceleration to I don't know if there is a way to enable rocm when running that build command. Also, the Basically the problem is that the upstream change here made it so that we need the rocm libraries as runtime dependencies instead of build time, since they aren't fully pulled in until runtime. I think anyway, I haven't completely tested this. Here is it working (at least to the point of seeing my AMD card): |
|
@volfyd I tested these changes with 0.1.37, but it doesn't appear to work for me; I'm not sure if I'm doing something wrong. My changes are in this branch. I did test with Ollama does run, but doesn't detect the libraries. This is the relevant section of ollama's debug logs. I also confirmed that only the cpu is used when actually running a model. |
|
@abysssol Maybe you need HSA_OVERRIDE_GFX_VERSION? I tried 0.1.37 and it is working for me. So I run the daemon here: and then in another window and then checking back with the logs in the first window: ... |
|
I made an override. It's really messy because overriding Go builds is really messy. |
|
@volfyd I tried setting Have you tested the changes on my branch to see if they work for you? If it doesn't work, then I must have just set something up wrong, but if it does work for you then there must be some strange hardware specific incompatibility. |
|
Your nixpkgs works for me if I use the HSA_OVERRIDE_GFX_VERSION="11.0.0" environment variable. (I omited some output): So then I got curious, and I removed the change I proposed (add the libraries as runtime dependencies). Note that the diff is relative to your commit, so it is showing me removing the runtime deps. And that still worked! So my theory may be incorrect. I don't know why this is working on my system and not yours. Perhaps I have something different in my NixOS config? I will try again, reverting even more of the change... |
|
OK, so here I am trying it with most of the "fixes" removed (except the build fix related to the I git reset to peel off your commit, and then unstage the bits I don't want, run git restore to get rid of them. Then the diff below is relative to a plain nixpkgs. And you can see that it still works! I should probably take ollama out of my NixOS config and see if that makes a difference. |
|
So in my NixOS config, I have If I remove that, I get Higher up @abysssol you said
I am not sure if "crash" means it exits with an error or without, but I have never seen it crash really. It either works if I run it with the environment variable and the necessary |
I seem to remember testing it a few weeks ago and it crashing without any output/logs/error, somewhat like a segfault or linker error (from unavailable libs). However, my current testing has never had this outcome either, so I'm not sure if it's a false memory, or if I confused crashing as an outcome of a different experiment as being the result of removing the lib check, or if a different ollama version had different behavior in this regard. You can probably safely ignore that comment, as it seems to be irrelevant/outdated/false. My apologies for the confusion. I tried adding the packages from your However, since this seems to work right for you it may work right for others, so it's probably worth putting these changes in whatever version ends up getting into unstable to at least allow correct functionality for some users. Could you try removing everything from your runtimeLibs = lib.optionals enableRocm
(rocmLibs ++ [
rocmPackages.rocm-smi
+ rocmPackages.clr.icd
+ rocmPackages.rocm-comgr
+ rocmPackages.rocm-runtime
+ pkgs.rocm-opencl-icd
+ pkgs.rocm-opencl-runtime
+ pkgs.libva
+ pkgs.vaapiVdpau
+ pkgs.libvdpau-va-gl
+ pkgs.amdvlk
])
++ lib.optionals enableCuda [
linuxPackages.nvidia_x11
];If that works, then try removing libraries to find the minimum necessary libraries for ollama to function correctly. That way someone can use ollama without having to first add anything into their Thank you for all your help with this. |
|
Here are the actual packages I need in my NixOS Putting them into the nixpkgs definition instead doesn't work I don't have an explanation |
|
I have an explanation! ollama looks for rocmhipblas and rocmlocblas and requires them to be in the same directory. They can be in the same directory if they are installed by the NixOS system, but otherwise they will each be in their own directory. https://github.com/ollama/ollama/blob/5bece945090b94a3f1eab03be48fb6f6b25e1e79/gpu/amd_linux.go#L33 https://github.com/ollama/ollama/blob/main/gpu/amd_common.go#L14 |
|
One thing to note. If I make the check (the check for hipblas and rocblas in the same directory) less strict, to only require that rocblas is found, then ollama appears to work with the graphics card. ollama does not appear to explicitly load hipblas at runtime (but it does use it at build time) so I think the check in the code is just overly strict. |
|
I've finally managed to figure out how to get the rocm build to work for me, all thanks to you @volfyd! I've created pr #312608 for further testing and discussion. If all goes well (rocm works correctly for others and no further issues are found), I want to merge it as soon as the next stable nixos version is released. |
Description of changes
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.